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ABSTRACT 

Sub-image search with high accuracy in natural images still re- 
mains a challenging problem. This paper proposes a new feature 
vector called profile for a keypoint in a bag of visual words model 
of an image. The profile of a keypoint captures the spatial geome- 
try of all the other keypoints in an image with respect to itself, and 
is very effective in discriminating true matches from false matches. 
Sub-image search using profiles is a single-phase process requiring 
no geometric validation, yields high precision on natural images, 
and works well on small visual codebook. The proposed search 
technique differs from traditional methods that first generate a set 
of candidates disregarding spatial information and then verify them 
geometrically. Conventional methods also use large codebooks. 
We achieve a precision of 8 1 % on a combined data set of synthetic 
and real natural images using a codebook size of 500 for top- 10 
queries; that is 31% higher than the conventional candidate gener- 
ation approach. 

1. INTRODUCTION AND MOTIVATION 

Community contributed image sites (e.g. Flickr.com) and stock 
photo sites (e.g. Gettyimages.com) are witnessing an unprecedented 
growth in the recent decade. Search of images by example is one 
of the most common tasks performed on these data sets. A related 
task is sub-image retrieval 1 15., 18 10]. It extends traditional full- 
image search by allowing users to select a region in an image and 
then, search for similar regions in a repository. Sub-image search is 
a very important tool for harnessing the potential of ever growing 
image repositories. Sub-image search as a tool is also being incor- 
porated in online shopping (e.g. like.com) to help customers search 
products using images. It is also an important tool for analysis and 
study of biological and medical image datasets. 

Community contributed repositories contain images of scenes 
or objects taken under varying imaging conditions. These images 
have affine, viewpoint, and photometric differences and also vary- 
ing degrees of occlusion. Local descriptors like SIFT (9j[T4][T3][T2) 
are used in literature (15] [21] 1 6 19] with fair success to measure 
similarity between natural images. Images are scanned to detect 
keypoints |5 |, covariant regions are extracted around each point, 
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Figure 1: A pathological case for sub-image search using bag of 
words model and L p norm. False match (b) is obtained in the 
top-5 results for query image (a) because spatial relationship is 
not considered. 



and finally a high dimensional local feature vector |9| represen- 
tation of each region is obtained |T4| |T2| . We know the spatial 
location, geometry of the covariant region, and high dimensional 
feature vector representation of each keypoint detected in an im- 
age. Thus, each image is transformed into a bag of feature vectors. 

Researchers have pursued two-phase techniques to retrieve sim- 
ilar images using local descriptors. The first phase consists of can- 
didate generation disregarding spatial relationships between key- 
points and the second phase consists of geometric verification. Can- 
didates are generated using two common approaches. One ap- 
proach 1 15 19] transforms each image into an orderless bag of 
visual words. Feature vectors of all the images are clustered and 
each feature vector is assigned the symbol of its cluster called vi- 
sual word. Thus, all the feature vectors of an image are transformed 
into visual words and an image is finally represented as a histogram 
of these visual words. This enables leveraging and extending ex- 
isting text-retrieval techniques to image search. Distance between 
image histograms is computed using L p norm. A given query im- 
age is also transformed into a histogram of visual words and its 
top-k candidates are retrieved. Another approach (6][9j finds top-k 
nearest neighbors of each keypoint feature vector of the query im- 
age in the dataset of local feature vectors of all the images. It ranks 
images based on the total number of nearest neighbors detected in 
them and retains top-k images as candidates. Both approaches are 
made efficient by using indexing techniques (l][4). 

We present two pathological results obtained by sub-image search 
using the first-phase methods as described previously. Figure p^a)] 
is a query sub-image and Figure [T(b)| is an image in the top-5 re- 
sults retrieved from the dataset. We can see that the components of 
the query pattern are scattered randomly in Figure [1(b)] and it is not 
a meaningful match. A local descriptor like SIFT is computed just 
using neighborhood pixels around a keypoint. Similar keypoints 
will be computed in both the query and the given result image be- 
cause result image contains all the components of the query. The 
only way to distinguish between these two images is to consider 
the spatial layout of the keypoints. These methods generate this 
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Figure 2: A pathological case for sub-image search using bag of 
words model and L p norm. True match (b) is not retrieved be- 
cause sub-image query (a) is a very small fraction of the actual 
image (b). 



false match because they do not take spatial relationships between 
keypoints into account. Figure |2(a)| is another query sub-image 
and Figure [2(b)] is a true match not returned in the top-5 results. 
An explanation of this anomaly is that the query region is a very 
small fraction of the whole image and therefore, random outliers 
score better than the true match when similarity is computed just 
using orderless bag of words representation of images. Both these 
examples motivate the necessity of using spatial relationships be- 
tween matched points in a sub-image search to get high accuracy. 
Existing literature uses geometrical verification (Hough |9|, Ran- 
sack 1 15 6 16 1) on the candidate images to find the best match or 
re-rank the top-A; candidates. Generating a small set of high quality 
candidates is very important for all the conventional approaches to 
reduce the high cost of geometric verification. Geometric verifica- 
tion algorithms are iterative in nature and costly. 

Sub-image search with high accuracy is very challenging and 
still remains an open problem. There is need of further develop- 
ment of better techniques to retrieve high quality images. Success 
of sub-image search tools in optimally utilizing image repositories 
for various applications is directly constrained by the accuracy of 
their search results. The focus of our work is to develop a feature 
vector and a sub-image similarity measure using this feature vector 
that yields high accuracy for sub-image search. 

In this paper, we present a new feature vector called profile. We 
construct a concentric profile for each keypoint in a bag of visual 
words representation of an image (Section [3J. A keypoint' s pro- 
file approximately captures the spatial relationships of other key- 
points in the image with respect to itself. Sub-image retrieval for 
a given query is a single-phase search of the best matching profile 
in the database (Section [4). Our feature produces high quality re- 
sults for sub-image search without geometric verification using a 
single- step search and a small visual codebook. We perform exper- 
iments to empirically validate our method in Section [5] We show 
that our profile achieves comparatively higher accuracy on the land- 
mark dataset provided by Philbin et al. 1 15 1. We also prepared a 
dataset which is a collection of natural and synthetic images. We 
used a small codebook of size 500 for bag of words representation 
of images compared to thousands or even a million fT5) used in 
the literature. A conventional user mostly views top-k results of a 
query | 20 1 returned by a search engine. Therefore, high accuracy 
in top-k results is really important. We specifically focus on empir- 



ically analyzing accuracy of top-k matching images retrieved using 
our feature vector. Our approach achieves 81% precision on our 
dataset for top-10 results using code book size of 500; this is 31% 
higher than the precision of conventional methods. 

The main contribution of our paper is the development of a ro- 
bust feature vector for each keypoint that captures the spatial re- 
lationship of keypoints in an image, and a measure of similarity 
between the feature vectors for retrieving similar images. 

2. RELATED WORK 

Svetlana et al. 1 7 1 proposed to construct a single feature vector 
for the whole image by concatenating histograms of local features 
of sub-regions obtained by repeatedly splitting the image at various 
scales in a principled way (Spatial Pyramid). They used it for full- 
image categorization. This feature is a global feature vector and is 
not directly designed for sub-image search. Feature vector may fail 
to find those matching sub-regions of database images for a given 
query which get split into multiple regions during feature vector 
construction. The similarity score of a sub-region of an image with 
its full-image using this feature vector may be small, depending on 
the similarity measure used, making it difficult for the true match to 
be distinguished from false matches. Further, this feature is neither 
rotation nor translation invariant because of the use of orthogonal 
split lines. Our profile is computed for each keypoint of an im- 
age and matches are retrieved based on keypoint profile similarity 
rather than full-image similarity. Therefore, our profile is naturally 
suited to sub-image search. We discuss scale, rotation, and transla- 
tion invariance of our profile in Section [3] 

Weijun et al. (21 1 proposed to spatially cluster local descriptors 
per image with a bound on the number of points in the cluster and 
its radius. They represented each cluster as a bag of visual words 
and each image as a collection of these clusters. Images were 
ranked based on the count of the clusters that were top-k nearest 
neighbors of the clusters of a given query image. They achieved 
a precision of 65%. Philbin et al. fT5) clustered the local descrip- 
tors to build a visual codebook of size 1 million and represented 
each image as a bag of visual words. They used the L 2 measure 
to find candidates and then validated those using LO-RANSACK 
for restricted sets of transformations. They reported mean average 
precision of 66% on specialized dataset of landmark images using 
1 million visual words and geometric verification. Yan et al. (6) 
proposed to generate candidates by nearest neighbor search using 
Locality Sensitive Hashing |4 1 and validated using RANSACK for 
sub-image retrieval for near duplicate image detection and copy- 
right protection. They experimented only on a synthetic dataset. 
Lowe et al. (5) performed nearest neighbor search using BBF algo- 
rithm [2 | and geometric validation by Hough transform to recog- 
nize objects in images. 

3. PROFILE CREATION 

In this section, we design our new feature vector called profile 
which is created for each keypoint in an image. A profile of a key- 
point is a structural representation of the spatial layout of all other 
keypoints around it. We assume that an image / has been prepro- 
cessed and transformed into an orderless bag of visual words. We 
also know the coordinate of each keypoint detected in an image. To 
form the profile, we draw concentric circles around each keypoint 
p of an image. Figure [3] shows concentric circles for keypoint a 
and keypoint b. Each ring is represented as a histogram h of visual 
words of keypoints lying in it. Profile H = (hi , /12, • • • , hm) of a 
keypoint p, where m is the number of rings, is a concatenated list 
of ring histograms ordered from the center towards the outer rings. 



Figure 3: An image with its keypoints feature vectors represented as visual words. Concentric circles are drawn around keypoint a 
and keypoint b. The profile of a keypoint is defined by a set of histograms. Each histogram summarizes the keypoints in a ring. The 
number of keypoints in a ring doubles as we move outwards from the center. 



The dimension of a profile is directly proportional to the codebook 
size and the number of rings around a keypoint. The number of 
points in a ring (called size) increases as we move away from the 
center, and is defined by Size(hi) = 2 * Size(hi-i). The number 
of points in the first ring is no which is user defined and is same for 
every keypoint's profile across all the images. The ith ring contains 
2 t ~ 1 no points except in the case of the last ring where the required 
number of points may not exist. It's worth noting that we do not fix 
the radii of the rings but fix the number of points in a ring, which is 
a function of no. As a result, number of rings in the profiles of all 
the keypoints in a given image will be the same but the radii of the 
rings may vary. Radii of the rings are based on the spatial density 
of other keypoints around a given keypoint. An image having more 
keypoints will have more rings than an image with less keypoints 
irrespective of the size of the images. Rings of an image with a high 
spatial density of keypoints will have narrower rings compared to 
an image having a low spatial density of keypoints. The profile of 
each keypoint of an image will differ depending on its position in 
the image because keypoints captured in the rings of different pro- 
files will be different. 

Similarity between profiles: The similarity between two pro- 
files Hi and Hj , where the number of rings in Hi is m and in Hj 
is n respectively, is given by 

min(m,n) 

Sim{Hi,Hj) = e " Afe *S/c 

fc = 

S k = Sim(hk(Hi),hk(Hj)) 

Here, A is a decaying parameter learned empirically and is always 
positive. The similarity between corresponding ring histograms 
(Sk) can be computed using Jaccard's Coefficient or Cosine mea- 
sure. The distance between two profile is computed as a comple- 
ment of their similarity. For the purpose of explanation, assume 
Jaccard's Coefficient as the measure of similarity. For this mea- 
sure, the maximum value of similarity between two corresponding 



histograms is 1. Therefore, the best value of similarity between two 
profiles is 

min(m,n) 

Sirrimax(Hi,Hj) = ^ e~ Xk * 1 

fc=0 

The distance between two profiles is given by 

D(Hi, Hj) = Sim rnax (H i ,H j )-Sim(H i ,Hj) 

min(m,n) 

= £ e- xk *(l-S k ) 
This can be represented as a recurrence 

min(m,n) 

D(H^H 3 ) = D(Hi, Hj)i + Yl e~ Xk ^(l-S k ) 

fc=(Z+l) 

where D(Hi,Hj)i is the distance computed for the first / cor- 
responding ring histograms. Other similarity measures and their 
complements can be used to find similarity and distance between 
two profiles respectively. We found Jaccard's Coefficient as a mea- 
sure of similarity between histograms to perform better than other 
methods empirically. 

The similarity in the proximity of a keypoint should be weighed 
higher but should be weighed less as we move away to decimate 
the effect of noise and overfitting in matching. This is the reason 
for the exponentially decaying aggregation of similarity between 
corresponding rings of profiles and keeping increasingly more key- 
points in the rings away from the center. 

A keypoint is localized in an image and its local descriptor cap- 
tures the property of its small neighborhood in the image. Local- 
ity of a keypoint makes it naturally fit for sub-image search. Our 
concentric profile captures spatial layout of keypoints of the whole 
image with respect to a given keypoint. The profile of a keypoint 
gives it the structured global view of the whole image making it se- 
mantically richer than the keypoint itself. Therefore, the profile of a 




Query Image Q Database Image 1: 90° Rotation of 
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Figure 4: The profile of keypoint a in query image Q is same as 
its corresponding profile in r otated image I. 




Database Image I: Scaled Version of 
Query Image Q 



Figure 5: The profile of keypoint a in image Q and image I re- 
main the same in spite of scaling. The ring radius has increased 
in image I with an increase in scale. 

keypoint can distinguish between a true match and a false match ef- 
fectively. A sub-image search based on profiles would have higher 
accuracy than bag of visual words based search. 

Profile Vector Robustness: Visual words which form our pro- 
file's histogram are obtained from local descriptors. It should be 
noted that although local descriptors like SIFT provide invariance 
to affine changes for similarity search, feature vectors created by 
spatial division of image around a keypoint may compromise in- 
variance to affine changes. Our profile vector retains invariance 
to rotation, scaling, translation, and occlusion. We perform image 
division only in a radial direction. Therefore, our method yields 
same concentric profile for a given keypoint irrespective of the ro- 
tation of an image making it rotation invariant. We see in Figure [4] 
that the profile of keypoint a in query image Q remains same in 
rotated image /. The scaling of an image alters the relative dis- 
tance between points, as seen in image / of Figure [5] Therefore, 
a fixed radius concentric division will fail to provide scale invari- 
ance even though the local descriptors are scale invariant. We keep 
equal number of keypoints in rings while constructing the profile 
of a given keypoint irrespective of the size and scale of an image. 
This technique generates the same profile for a given keypoint in 
two images irrespective of the scale. The profile of keypoint a of 
image Q in Figure [5] is the same as the profile of keypoint a in the 
scaled image /. Therefore, our profile feature remains invariant to 
scaling. Our search methodology automatically preserves transla- 
tion invariance and provides robustness to occlusion as discussed 
in Section [4] 

4. SUB-IMAGE SEARCH 

In this section, we discuss the method used to compute the sim- 
ilarity between a pair of profiles. This similarity is used for sub- 
image search. Similar keypoints will be extracted from a query 




Query Image Q. Database Image I: Translation of image 

Q with additional objects. 



Figure 6: Keypoint a is translated in image I. The profile of 
keypoint a in image I has high similarity to its profile in image 

Q. 

image Q and a matching sub-region of a database image I by local 
descriptor methods. Therefore, the keypoints captured in the more 
weighted histograms of the profile of a keypoint q in a query im- 
age Q would be same to the profile of a keypoint p in the matching 
sub-region of a database image /. This will give high similarity 
score between the profiles. We exploit this property to retrieve best 
matching sub-regions from the database. All the images, repre- 
sented as bags of words, are converted into bags of profiles. Query 
Q is also processed to generate a bag of profiles. We search the 
best matching profile pi in a database for each query profile qj and 
finally, choose the highest scoring pair (qj, pi) among these. The 
sub-region around the keypoint pi of the highest scoring pair is the 
best matching image. If Q has m profiles and N is the total num- 
ber of profiles in the dataset then the best matching sub-image is 
the region around ith profile where i is obtained by 

arg • max < max Sim(H q . , H„. ) > 

j<m i<N J J 

Our search algorithm inherently provides translation invariance be- 
cause it searches for the best matching profile. We create a pro- 
file for each keypoint of an image. Since, local descriptors are 
translation invariant, relatively similar keypoints will be detected 
in a matching sub-region of a translated image and the given query. 
Therefore, profiles of the query will have high similarity with the 
profiles of the keypoints detected in the matching sub-region com- 
pared to the profiles of random keypoints and our algorithm will 
successfully find these translated matches. Image / in Figure[6]has 
translation of few objects in image Q as well as additional new ob- 
jects. We see that the concentric profile of keypoint a in image / 
is very similar to the corresponding profile of a in image Q. Our 
algorithm will also find a partially occluded but true matching sub- 
region of a database. Profile of a keypoint detected in the preserved 
part of the matching sub-region will have relatively high similar- 
ity score with some keypoint profile of the given query. Profile of 
keypoint a in image / of Figure [7] will have high similarity with 
the profile of keypoint a in image Q. Therefore, the true matching 
sub-region will rank higher on similarity score compared to random 
profiles. 

5. EXPERIMENTAL EVALUATION 

In this section, we present a comparative study with a state of 
art technique using the author's dataset. Next, we describe our own 
dataset and perform experiments to show that our profile based ap- 
proach yields high precision for sub-image search with a small vi- 
sual codebook. 

5.1 Comparative Evaluation 
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Database Image I which is 
Image Q. after Occlusion 



Figure 7: The profile of key point a in image I has high similar- 
ity with profile of a in image Q even after occlusion. 
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Table 1: Comparative precision for 36 Oxford building queries 
for varying k. 

We applied our profile feature based search on the Oxford dataset 
provided by Philbin et alQ [15]. This dataset has bag of visual 
words representation of 5,062 high resolution (1024 x 768) im- 
ages of 1 1 different Oxford buildings collected from Flickr. It has 
55 queries with ground truth for comparative validation. Philbin et 
al. ITS | used the L 2 distance on bag of words model to rank all the 
images for each query and then computed the average precision. 
They reported mean average precision of 64.5% without geomet- 
ric validation and mean average precision of 66% with geometric 
validation. We chose the top-36 most challenging queries which 
were better representative of sub-image search as seen in Figure [8] 
and specifically left out the queries which were more of a full im- 
age search. For a given k, we retrieved top-k similar images for 
each of the 36 queries from this dataset. We computed the ratio of 
true results retrieved to the total images retrieved (precision). We 
averaged the precision over all the 36 queries. We also retrieved 
top-k results for each query by just computing the L 2 distance on 
bag of words model as proposed by Philbin et al. and computed 
mean precision. We present the comparative percentage precision 
in Table[T]for varying k from 1 to 10. For a profile based search, we 
can see that the top- 5 results have mean precision as high as 94% 
and it drops to only 86% for top- 10 results. For a bag of words 
based search, mean precision drops to 77% for top- 10 results. A 
typical user generally looks at top few results of a query returned 
by a search engine. We see that our profile based feature vector 
returns highly accurate results for challenging queries on the land- 
mark images in top- 10 compared to the state of the art method. 

5.2 Dataset Preparation 

1 http://www.robots.ox.ac.uk/ vgg/data/oxbuildings/index.html 
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Figure 8: 36 sub-image queries marked with yellow box from 
Oxford Dataset. 

We downloaded natural images from Flickr. We also manually 
photographed a large number of scenes under varying conditions. 
We chose 35 random queries from this real dataset of natural im- 
ages for our experiments. We synthetically created some test im- 
ages to put our method to even tougher challenges. We randomly 
chose 17 sub-images from the natural images and embedded them 
into other large natural images. We also added varying noise lev- 
els to these images as discussed in [11]. Some of the operations 
to add noise were rotation, scaling, shearing, gaussian blur, and 
averaging noise. We also split a sub-image and scattered its frag- 
ments into other images. Finally, we obtained a dataset of 1000 
images (800-natural and 200-synthetic). A snapshot of the dataset 
is shown in Figure[9] We used a total of 52 queries (35-natural and 
17- synthetic). 

Next, we created bag of words representation of each image. We 
extracted covariant regions 1 14] from each image and the corre- 
sponding 128 dimensional SIFT vector. We clustered the feature 
vectors by picking 500 random centers using k-means 1 8 1 cluster- 
ing. We chose k-means clustering because it is not only simple but 
embarrassingly parallelizable. A naive computation of k-means has 
complexity of 0(mnt) where m is the number of cluster centers, 
n is number of data points, and t is the number of iteration for 
k-means convergence. This can be reduced by using metric prop- 
erty 1 3 1. We repeated this process 10 times to choose the cluster set 
which has the minimum average sum of the squared error or scat- 
ter. We assigned the symbol to each cluster. We mapped each SIFT 
feature vector to the cluster symbol to which it belonged. This gave 
us a bag of words representation of an image. For creating profiles, 
we chose no=50 points in the first ring and used A=l/3 as decay- 
ing parameter for aggregation. We found the Jaccard's Coefficient 
to yield better results than other distance measures and used it to 
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Figure 9: A snapshot of our experimental dataset which has 
many variations and is a close representative of a natural image 
dataset. 

compute similarities between ring histograms of our profiles. 

5.3 Precision Test 

We computed precision of top-k results obtained using our pro- 
file based search and compared it against the precision of conven- 
tional methods which compute similarity only on the bag of words 
without taking spatial relationships into account. We used Cosine, 
Li, and Jaccard's Coefficient as the measure of similarity for con- 
ventional methods. We considered both the schemes of standard tf- 
idf weighting 1 17] and without tf-idf weighting of visual words for 
the candidate generation approach. In a tf-idf approach, commonly 
occurring visual words are weighed less as they are less discrim- 
inative. We did not consider the tf-idf weighting of visual words 
for our profile based approach. Figure [T0| and Figure [TT] show the 
comparative precision of various methods for the non-weighted and 
the weighted cases, respectively. We find that our approach yields 
81% precision rate for top- 10 results. Without tf-idf weighting, 
conventional method with cosine measure gives the best precision 
of 50% which is 31% less than profile based method. With tf-idf 
weighting, conventional method with Jaccard's coefficient gives the 
best precision of 39% for top- 10 results. Search with tf-idf weight- 
ing achieves less precision compared to the search without tf-idf 
weighting for conventional methods. We also experimented by 
varying A for our profile similarity and achieved more than 20% 
higher precision for every A < 1 compared to conventional meth- 
ods. We also experimented by weighting the symbols with the area 
of covariant regions but achieved less precision. We see that our 
method achieves higher precision than conventional methods used 
for candidate generation on a small codebook size and without any 
geometric verification. 

5.4 Visual Results 

We present the top-5 visual results for 4 real queries from our 
search in Figure[l2] Our profile based approach retrieves high qual- 
ity results irrespective of the kind of noise present in the dataset. 
We outline the matching sub-region in a result image with red box. 
We got all true matches in the top-5 results for the 3rd query which 
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Figure 10: Comparison of conventional methods, using dif- 
ferent distance measures without tf-idf weighting, with profile 
based approach. 
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Figure 11: Comparison of conventional methods, using differ- 
ent distance measures and tf-idf weighting, with profile based 
approach. 

got pathological result using conventional methods shown in Fig- 
ured We also verified for the 2nd pathological case shown in Fig- 
ure]^ and found that the result was returned in top- 10 results and 
all the better scoring images were true matches. 

6. CONCLUSION 

In this paper, we developed a simple but effective profile based 
feature vector that captures the spatial relationships between key- 
points of an image. Inclusion of spatial layout in feature vector 
improves the search result dramatically without the need of geo- 
metric verification. We proposed a method to measure the simi- 
larity between profiles. Our technique produced higher accuracy 
on landmark images compared to the state of the art method. We 
evaluated our technique on a mixture of synthetic and real natural 
images dataset over 52 queries and obtained a 81% precision for 
top-10 matches using a small code book size of 500. This was 31% 
higher than the conventional methods. 
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